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Approximate  Solutions  for  Certain  Optimal  Stopping  Problems 

Albert  John  Petkau 


1.  Introduction 


The  following  optimal  stopping  problem  (whic±i  is  one  of  several  dif- 
ferent problems  ifAiich  have  come  to  be  known  as  the  one-armed  bandit  problem) 
has  arisen  in  a number  of  statistical  applications  (Chernoff  and  Ray 
(1965 Chernoff  (1967 Mallik  (1971)):  Let  X(t)  be  a Wiener  process 
described  by  E[d:{(t)]  = p-dt  and  Var[dX(t)]  = cr  -dt  where  cr  is 
presumed  known.  One  is  permitted  to  stop  observing  the  process  X(t) 

at  any  time  t,  ^ < t < N,  and  receive  a payoff  X(t).  The  unknown 

2 

parameter  p is  assumed  to  have  a W(pQ,crQ)  prior.  What  is  the  optimal 
stopping  procedure? 

It  is  easy  to  verify  that  the  posterior  distribution  of  p given 
X(t'  ),  0 < t'  < t,  is  N(Y*(s*),s*)  where 


(1.1) 


Y*(s*)  = 


PqCt"^  + X(t)cr“^ 

-2  ^ . -2 
cr  + ta 


and 


(1.2) 


s*  = (o-‘ 


-2^-l 


ter  “ ) 


2 -2 

Here  s*  varies  from  s*  = cr^  to  s*  = (cr^ 
process  Y*(s*)  is  a Wiener  process  (in  the 


-2 

+ Ncr  ) . Furthermore,  the 

-s*  scale)  described  by 
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E[dY*(s*)]  = 0 ajid  Var[dY*(s*)]  = -ds*,  starting  from  Y*(s*)  = . 

The  loss  upon  stopping  at  (Y*(s*),s*)  is  -X(t)  vdiieh  from  (l.l)  is  a 

linear  function  of  -Y*(s*)/s*.  Applying  the  transformation  s = s*/s*^ 

Y(s)  = Y*(s'^)/s*^^^  lead-s  to  a normalized  version  of  this  stopping  problem 

where  s varies  from  s_  = cr„(o-  + Nor  ) to  s,  = 1 and  in  which  the 

0 0 0 1 

stopping  cost  is  given  by  d(y,s)  defined  by 

(lo3)  d(y,s)  = -y/s 

for  s > 1 with  stopping  enforced  at  s = 1. 

This  normal-ized  problem  is  a special  case  of  the  following  optimal 
stopping  problem:  Given  a Wiener  process  (Y(s),  s > s^}  in  the  -s 
scale  described  by  E[dY(s)]  = 0 and  Var[dY(s)]  = -ds  and  starting  at 
Y(Sq)  = y^,  find  the  stopping  time  S to  minimize  E[d(Y(S),S)].  If  we 
define  p(y^,Sq)  = inf  b(yQ,SQ)  \4iere  b(yQ,SQ)  is  the  risk  associated 
with  a particular  stopping  time  and  the  infimum  is  taken  over  all  such 
stopping  times,  p(y,s)  represents  the  best  that  can  be  achieved  once 
(y,s)  has  been  reached,  irrespective  of  how  it  was  reached.  An  optimal 
procedure  is  then  described  by  the  continuation  set  [(y,s):  p(y,s)  < d(y,s)). 

Chernoff  (1968)  has  demonstrated  that  one  should  expect  the  solution  (p,  ^ ) 
of  the  stopping  problem  to  be  a solution  of  the  following  free  boundary  problem; 


— p (y,  s ) 

= p^(y^s) 

for 

(y^s) 

e (p 

(l.M 

= d(y,s) 

for 

(y^s) 

e 

Py(yyS) 

= dy(y,s) 

for 

(y^s) 

e 
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Furthermore^  for  any  such  stopping  problem.  Van  Moerbeke  (1974)  has  shown 
that  one  sho\ild  never  stop  at  points  (y,s)  at  which  ^ d (y^s)-d  (y, s)  < 0. 

^ yy  s 

Applying  this  criterion  to  the  normalized  version  of  the  stopping 
problan  described  above,  hereinafter  referred  to  as  the  one-armed  bandit 
problem,  one  finds  that  {(y,s):  y > 0,  s > l}  is  a subset  of  the  optimal 
continuation  region  ^ . Chernoff  and  Ray  (19^5 ) have  shown  that  for  this 
problem  ^ can  be  described  as  ^ = {(y,s):  y > y(s),  s > 1}  and  have 
determined  asymptotic  expansions  for  the  boundary  curve  y(s)  in  the  regions 
of  large  s and  s close  to  1.  Ihe  leading  terms  in  these  expansions  are 
given  by 

~ 1 /2 

y(s)  ~ -(2sins)  ' as  s « 

y(s)  ~ -0.64  (s-l)^'^^  as  s -♦  1 . 

The  scale  z = -y/s  and  t = l/s  is  more  appropriate  for  applications 
and  these  expansions  are  illustrated  in  this  scale  in  Chernoff  and  Ray  (I965), 
It  is  evident  from  this  illustration  that  these  asymptotic  expansions 
are  inadequate  as  a complete  description  of  the  optimal  continuation  region. 
An  approximation  to  the  optimal  continuation  region  is  required  as  a 
description  of  the  optimal  procedvire  in  the  region  where  the  asymptotic 
expansions  are  clearly  Inadequate. 

Althou^  it  is  possible  that  refined  methods  of  asymptotic  analysis 
couild  lea,d  to  expansions  which  would  provide  an  adequate  description  of 
the  optimal  procedure,  the  purpose  of  the  present  paper  is  to  describe 
simple  methods  which  lead  to  arbitrarily  accurate  numerical  approximations  to 
the  optimal  continuation  region  for  the  one-armed  bandit  problem.  Although 
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most  of  the  discussion  in  the  present  paper  will  concentrate  on  the  one- 
armed  bandit  problem,  these  same  methods  could  be  applied  with  equal 
facility  to  any  optimal  stopping  problem  of  the  general  form  described 
above, 

2<>  An  Analogous  Discrete  Problem 

Consider  the  process  Y’ (s’ ) vtoich  starts  at  Y' (l+n-A)  = y'  and 
is  defined  by  Y'  (s’-a)  = Y’  (s’  ) + each  with  probability  ^ . This 

process  is  observed  for  at  most  n successive  times  and  the  cost  associated 
with  stopping  the  process  at  any  point  (y’^s’ ) is  given  by  d(y’,s’ ) 
defined  by  (1.3  )•  The  problem  is  to  find  a stopping  time  to  minimize  the 
expected  loss.  We  shall  denote  the  optimal  expected  loss  by  p' (y’,s’ ). 

For  this  problem,  a backward  induction  algorithm  becomes 

(2,l)  p'(y',l+n.A)  = min(d(y' ,1+n.A),  |■[p’  (y' +A^^^,l+(n-l)»A) 

+ P'  (y  -A^^^A+(n“l>A)]}  ■ 

for  n > 1 with  . p'  (y’  ,l)  = d(y’  ,l).  It  is  easy  to  verify  using  the 
methods  of  Chernoff  and  Petkau  (1976)  that  the  optimal  stopping  set  can 
be  described  as  {(y’,l+n.A);  y'  n > 1}  V7here  for  each  fixed 

value  of  A^  . {y^(A)}  is  a non-increasing  non-positive  sequence.  Note 
that  this  set  does  not  depend  upon  the  initial  point.  Further  note 
that  direct  application  of  (2.1)  yields  Y^Ca)  = 0 . 

Since  Y' (s’  ) is  a process  of  independent  increments  with  mean  zero 
and  variance  one  per  unit  change  in  -s’,  any  stopping  problem  for  the 
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Wiener  process  Y(s)  of  the  previous  section  can  be  imitated  by  the  use 
of  a small  value  of  A in  the  Y*  (s’  ) process.  As  A approaches  zero, 
the  solution  of  the  analogous  discrete  problem  would  be  expected  to 
converge  to  the  solution  of  the  Wiener  process  problem.  In  particular, 
for  the  one-armed  bandit  problem  this  leads  to  the  initial  approximation 

(2.2)  y(l+n*A)  « y^(A) 

where  y(l+n.A)  denotes  the  optimal,  boundary  for  the  one-armed  bandit 
problem  evaluated  at  s = 1+n-A  . 

It  remains  to  evaluate  the  sequence  [y^(A)).  Consider  the  Y' (s' ) 
process  defined  as  above  on  the  grid  of  points  { (y' ,s' ):  s' = l+n*A  , 
y'  = c+k*A^^^j  n = 0,1,2, ...  ,k=0,  +1,  +2,...)  . Note  that  the  grid  is 

1/2 

completely  specified  by  the  parameter  c(for  convenience,  assume  0 < c < A ' ). 
Examination  of  (2.1)  with  the  particular  form  of  d(y,  s)  given  in  (1.3) 
reveals  that  for  any  given  choice  of  A,  if  the  points  ((y', 1+n-A):  y'  < y*) 
are  stopping  points  then  so  are  the  points  ( (y' ,l+(n+l).A):  y'  < y*-A  ). 

This  observation,  together  with  the  fact  that  the  sequence  (y^(A))  is  non- 
increasing, implies  that  when  using  the  backward  induction  algorithm  (2.1) 
to  classify  the  grid  points  as  either  stopping  or  continuation  points,  the 
comparisons  implied  by  (2.1)  need  be  carried  out  at  only  a single  value  of 
y'  for  each  fixed  value  of  s' . The  algorithm  (2.l)  can  now  be  easily  imple- 
mented in  a direct  fashion. 

Due  to  the  special  nature  of  the  one-armed  bandit  problem,  namely 
the  fact  that  all  points  (y,s)  with  y > 0 and  s > 1 are  continuation 
points,  one  might  expect  to  be  able  to  improve  somewhat  upon  the  naive 
approach  outlined  above.  Consider  a particular  path  of  the  Y' (s' ) process 


5 


l/2 

originating  at  the  point  (yS^'  ) = (c+2«A  ' , l+n»A).  The  path  of  the 

1/2 

Y'  (s'  ) process  could  hit  the  line  y'  = c+A  for  the  first  time  at 
s'  = l+(n“l)<>A,  l+(n-3  )*Aj, . • . . Alternately^  the  path  could  remain  above 

1/2 

the  line  y'  = c+A  all  the  way  to  s'  = 1.  Noting  that  the  points 
(c+A^^^>  s'  ) are  continuation  points  for  all  s'  > 1 (since  y^(A)  = 0 and 
the  sequence  yj^(A)  is  non-increasing)  leads  to  the  relation 


(2-3) 


p'  (c+2<>A^^^,l+n.A)  = 51  (c+A^'^^^l+ (n-m)*A) 

m=l 


n+1 

Z j^d(c+(k+l).A^/",l) 

k=l  ^ 


where  p^  is  the  probability  that  an  ordinary  random  walk  starting  at 
0 first  passes  throu^  1 at  time  m and  <3^  is  the  probability 
that  an  ordinary  random  walk  starting  at  0 stays  above  -1  until  time 
n and  achieves  level  k-1  at  time  n.  From  Feller  (1968,  p.  89^ 
Theorem  2 ) one  finds 


(2.4) 


for  m even 


for  m odd  . 


In  addition  we  have  the  recursive  relation  p r P with  p^  = 

■^m+2  m+3  m - -"^l 

and  Pg  = 0.  From  Feller  (1968,  p.  75 Ballot  Theorem)  one  also  finds 
that 
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HlOl 


/ 

0 

for 

n even  and  k 

odd 

(2.5) 

q , = j 

n,k  ^ 

1 

k 

0 

! ""i  1 

for 

n odd  and  k 

even 

L n+1 

• 1 

1 n+k+1 j 

• 2 

otherwise  . 

\ 

^ 2 / 

The  relation  (2.3)  provides  a modified  method  of  carrying  out  the 
backward  induction  #iich  we  shall  call  the  truncation  method;  At  s'  = 1, 
the  risks  are  specified  by  d(y,s).  At  any  stage  s*  = l+n*A,  compute 

1/2 

the  risk  at  y'  - c+2»A  ' by  means  of  (2.3).  The  risks  at  the  levels 

1/2 

y'  = c+k«A  for  k = 1,0, -1, -2, . . . are  computed  using  the  algorithm 
(2.1)  in  the  fashion  described  above. 

Returning  for  a moment  to  the  one-armed  bandit  problem,  it  is  well- 
known  (and  obvious  from  (1.4))  that  changing  the  stopping  cost  function 
d(y,s)  by  adding  to  it  any  solution  of  the  heat  equation  leaves  the 
optimal  continuation  region  unchanged.  For  the  present  purposes,  it  is 
convenient  to  consider  the  new  stopping  cost  function  d^(y, s)  defined 
by  d.Q(ys,s)  = d(y,s)  + y,  that  is, 

(2.6)  ^^(y^s)  = y - y/s 

for  s > 1 with  stopping  enforced  at  s = 1.  Note  that  dQ(y,l)  s 0 . 
Denoting  the  corresponding  optimal  risk  by  PQ(y^s),  the  algorithm  (2.1) 
becomes 
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(2„7)  p^(y’A+n»A)  = min[dQ(y' ,1+n.A)^  |■[pQ(y' a) 

+ P^(y'"A^/^A+(n™l).A)]} 

for  n > 1 with  P^^iy'  }l)  = 0,  Further^  the  relation  (2.3)  becomes 

(2.8)  p'  (e+2*A^'^^,l+n»A)  = E Pj^-pQ(c+A^'^^;l+(n-m)eA)  . 

m=l 

This  reduces  the  computation  involved  in  carrying  out  the  truncation  method. 

To  this  point  we  have  simply  described  the  direct  a,nd  truncation  methods 
of  car’rylng  out  the  backward  induction  algorithm  for  the  Y'  (s'  ) process 
when  the  motion  of  the  process  is  restricted  to  a grid  specified  by  a 
parameter  c.  Implementing  the  algorithm  for  a sequence  of  c values 
(with  the  same  fixed  value  of  A)  allows  the  sequence  {yj^(A)}  to  be 
determined  to  within  any  desired  degree  of  accuracy. 

3.  A Refined  Approximation 

In  the  previous  section  we  have  presented  simple  methods  of  obtaining 
initial  approximations  to  the  solutions  of  optimal  stopping  problems  for 
a zero  drift  standard  Wiener  process.  These  methods  involve  replacing  the 
Wiener  process  problem  by  an  analogous  discrete  problem  involving  dichotomous 
random  variables.  The  relation  of  the  solution  of  any  such  Wiener  process 
problem  to  the  solutions  of  an  entire  class  of  analogous  discrete  problems 
is  considered  in  Chernoff  and  Petkau  (1976).  A particular  result  of  that 
paper  is  the  following  simple  approximate  relation  between  the  optimal 
boundary  of  the  Wiener  process  and  the  optimal  boundary  for  the  particular 
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analogous  discrete  problem  described  in  the  previous  section 


(3.1)  y(l+n«A)  = y^(A)  + 0.5 

(the  sign  being  determined  so  as  to  make  the  continuation  region  for  the 
Wiener  process  problem  larger).  For  the  one-armed  bandit  problem,  this 
leads  to  the  following  refinement  of  (2.2) 

(3.2)  y(l+n.A)  ^ yn(^)  “ ‘^*5  • 

To  illustrate  the  accuracy  of  these  approximations,  it  would  be 
desirable  to  evaluate  these  approximations  in  a Wiener  process  problem 
for  which  the  exact  solution  is  known.  A normalized  version  of  a problem 
discussed  in  Van  Moerbeke  (197^)  is  the  following:  fX(s),  0 < s < 1}  is 
a zero  drift  standard  Wiener  process.  One  is  permitted  to  stop  the  process 
at  any  time  s,  0 < s < 1.  The  reward  for  stopping  the  process  at  the 
point  (x,s)  is  given  by  g(x,  s)  defined  by 

I l-s+2x^  for  X > 0 

(3.3) -  g(x,s)  = i 

[ 1-s  for  X < 0 . 

The  problem  is  to  find  a stopping  time  that  maximizes  the  expected  reward. 
Van  Moerbeke  (197^)  proves  that  the  optimal  continuation  region  for  this 
problem  can  be  described  as  [(x,s):  x > x(s),  0 < s < l)  where 

i>j . , . \l/2 

x(sj  = -a  (l-s  ) and  a is  the  solution  of  the  simple  equation 
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XOO  p 

exp[A.a-A.  /2]dA  = 1 which  can  easily  be  detemined  to  be 

0 

a 1 0.5061.  Modifying  the  reward  function  to  be  gix^s)  = g(x,s)-2[x^+l-s] 
does  not  change  the  solution  of  this  problem  and  it  is  easily  seen  that 
the  methods  of  the  previous  section  are  directly  applicable  (in  particular, 
g(x,l)  s 0 for  X > 0). 

This  Wiener  process  problem  has  been  approximated  by  three  different 

analogous  discrete  problems,  those  corresponding  to  A = 0.01,  0.0025  and 

0.000625.  In  addition,  for  each  fixed  value  of  A the  computations  were 

l/p 

carried  out  for  values  of  the  parameter  c varying  from  0 to  A ' in 
steps  of  0.001.  Thus  each  individual  member  of  each  of  the  three  sequences 
(x^(A))  is  located  to  within  an  error  of  0.001,  In  addition  the  corrected 
sequences  {x*(a)}  defined  by  x*(a)  = x^(a)-0. 5A^^^  were ' evaluated. 

These  six  approximating  sequences  and  the  exact  solution  x are  illustrated 
in  Figure  1 in  the  (x,s)  scale.  Here  x^  = {x^(O.Ol))  , 

Xg  = {x^(0.0025 )),  = {x^ (0, 000625 )}  and  similarly  x*  = [x*(0.0l)), 

^2  ~ ^ ~ {x* (0, 000625 )) . This  figure  clearly  illustrates 

that  for  this  particular  problem  whereas  the  approximations  provided  by 

Xj^,  Xp  and  x^  are  quite  crude,  the  approximations  provided  by  x*,  x* 

and  x^  (particularly  both  Xp  and  x^)  are  exceptionally  accurate, 

being  virtually  indistinguishable  from  each  other  and  from  the  exact 

solution.  The  fact  that  xj  is  not  too  accurate  reflects  the  fact  that 

* 

when  using  these  approximations,  one  must  begin  with  a reasonably  small 
value  of  A. 
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The  exceptional  performance  of  the  refined  approximation  (3.1)  in 
Van  Moerbeke' s problem  leads  us  to  hope  that  the  same  type  of  behavior 
will  occur  in  the  one-armed  bandit  problem.  In  order  to  examine  this 
possibility^  the  one-armed  bandit  problem  was  approximated  by  three  different 
analogous  discrete  problems,  those  corresponding  to  A = 1.0,  O.25  and 
0.0625.  For  each  fixed  value  of  A the  computations  were  carried  out  for 
the  region  1 < s < 100  and  for  values  of  the  parameter  c varying  from 

1/2 

0 to  A in  steps  of  0.01.  Thus  each  individual  member  of  each  of  the 
three  sequences  {y^(A)}  is  located  to  within  an  error  of  0.01.  In 
addition  the  corrected  sequences  {y^(A)3  defined  by  y*(A)  = y^(A)-0. 5A^^^ 
were  evaluated.  These  six  approximating  sequences  are  illustrated  in 
Figure  2.  Here  y^  = {y^(l.O)},  y^  = {yj^(0.25)},  y^  = {y^(0.0625))  and 
similarly  y^  = (y*(l.0)},  y*  = {y*(0.25)),  y*  = {y*(0.0625 )},  This  figure 
clearly  indicates  that  for  the  one-armed  bandit  problem  the  approximations 
provided  by  y^,  y*  and  y*  are  exceptionally  accurate,  these  curves  being 
indistinguishable  from  one  another. 

As  pointed  out  in  the  introduction,  for  applications  of  the  solution 
of  the  one-armed  bandit  problem,  the  (z,t)  scale  where  z = -y/s  and 
t = l/s  is  more  appropriate.  In  order  to  obtain  an  accuxate  approximation 
to  the  optimal  stopping  boundary  in  the  (z^t)  scale  in  as  efficient  a 
manner  as  possible,  the  computations  were  carried  out  as  follows;  Beginnirg 
with  a very  small  value  of  A,  the  boundary  was  approximated  in  a small 
interval  of  s in  the  manner  described  above.  Successively  larger  values 
of  A were  then  employed  to  approximate  the  boundary  in  successively 
larger  intervals  of  s.  These  approximations  to  the  optimal  boundary. 
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determined  in  overlapping  intervals  of  s,  were  then  superimposed  to 
obtain  the  final  approximation  to  the  optimal  boundary#  Since  the  values 
of  A used  were  chosen  in  such  a way  as  to  yield  the  desired  accuracy, 
only  the  value  c = 0 was  used  in  these  computations.  The  computations 
were  carried  out  using  both  the  direct  and  the  truncation  method.  The 
truncation  method  reduced  the  computation  time  required  by  a factor  of 
two.  The  resulting  approximation  to  the  optimal  stopping  boundary  and 
the  asymptotic  expansions  of  Chernoff  and  Ray  (1965)  are  illustrated  in 
Figure  3.  Here  and  z^  denote  the  boundaries  obtained  using  the 

asymptotic  expansions  for  t close  to  0 (s  large)  and  t close  to 
1 (s  close  to  1)  respectively  and  z denotes  the  boundary  obtained  by 
means  of  the  computations  described  above. 

Discussion 

The  optimal  stopping  boundary  for  the  one-armed  bandit  problem  has 

been  approximated  before  and  appears  in  the  literature  in  both  Mallik 

(1971)  and  Chernoff  (1972).  These  previous  approximations  were  determined 

without  the  benefit  of  the  "correction  for  continuity"  given  in  (3.1). 

The  purpose  of  the  present  paper  was  to  describe  explicitly  how  these 

boundaries  could  be  determined,  to  demonstrate  that  Exceptional  accuracy 

is  possible  and  to  indicate  that  these  computations  can  be  quite  efficient. 

Obtaining  the  present  approximation  to  the  boundary  of  the  one-armed  bandit 

problem  involved  ten  separate  runs,  the  i-th  run  approximating  the  boundary 

in  the  region  s = 1 to  s = 1+1000-A  using  a grid  determined  by 
_l|.  1 i -1 

A = 10  *4  , The  entire  computation,  approximating  the  boundary  in  the 
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region  1 < s < 25.'.000,  required  just  it-  seconds  of  computation  time  on 
the  IBM  370/168  at  U. B. C.  The  objective  in  the  present  computation  was 
to  obtain  an  accurate  approximation  in  the  (z,t)  scale.  Detailed 
examination  of  the  output  leads  to  the  empirical  estimate  that  in  the 
(Zft)  scale  the  boundary  has  been  located  to  within  an  error  of  about 
0„002. 

As  indicated  in  the  introduction^,  these  same  methods  could  be 
applied  with  equal  facility  to  any  optimal  stopping  problem  of  the  general 
form  described  there.  (in  Petkau  (1977)  these  methods  have  been  employed 
to  obtain  the  optimal  continuation  region  for  a stopping  problem  in  ^ich 
the  optimal  continuation  region  can  be  described  as  the  set 
{(y^s);  y^(s)  < y < y2(s)^  s > l]  where  y^(s)  / The  connection 

between  such  optimal  stopping  problems  and  free  boundary  problems  involving 
the  heat  equation  of  the  form  (l.^)  makes  it  clear  that  these  same  methods 
could  be  used  to  determine  numerical  solutions  of  such  free  boundary 
problems.  The  pz-oblem  of  obtaining  numerical  solutions  to  free  boundary 
problems  has  received  considerable  attention  in  the  literature  (see  for 
exam.ple  Sackett  (1971 ) and  Meyer  (1977  ))•  Whether  the  methods  proposed 
here  provide  a reasonable  alternative  to  these  more  general,  methods  is  a 
question  that  remains  to  be  answered. 
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